Method of performing communication load balancing with multi-teacher reinforcement learning, and an apparatus for the same

ABSTRACT

A server may be provided to obtain a load balancing artificial intelligence (AI) model for a plurality of base stations in a communication system. The server may obtain teacher models based on traffic data sets collected from the base stations, respectively; perform a policy rehearsal process including obtaining student models based on knowledge distillation from the teacher models, obtaining an ensemble student model by ensembling the student models, and obtaining a policy model by interacting with the ensemble student mode; provide the policy model to each of the base stations for a policy evaluation of the policy model; and based on a training continue signal being received from at least one of the base stations as a result of the policy evaluation, update the ensemble student model and the policy model by performing the policy rehearsal process on the student models.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to U.S. Provisional Patent Application No. 63/253,089, filed on Oct. 6,2021, in the U.S. Patent & Trademark Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a method of performing load balancing in acommunication system via multi-teacher reinforcement learning, and anapparatus for the same, and more particularly to a method for creating ageneralized control policy using multiple teacher networks and multiplestudent networks and performing traffic load balancing based on thegeneralized control policy, and an apparatus for the same.

2. Description of Related Art

Communication traffic load balancing is essential for the performance ofa mobile communication system, such as a fifth-generation (5G) or asixth-generation (6G) mobile communication system. In the real world,since communication traffic patterns dynamically change in real time andeach base station has limited resources, it is of critical importance todeploy resources as close to the actual demand as possible to maintainthe system performance and also to avoid waste of resources.

Reinforcement learning (RL), particularly deep RL, can achieve adequateperformance on different control tasks, such as traffic load balancingtasks. RL aims to learn an optimal control policy through interactionswith the environment of a communication system. Deep RL combines neuralnetworks with RL and further enables the RL agents to deal with morecomplex environments. However, deploying RL algorithms for real-worldproblems can be very challenging. Most online RL algorithms require alarge number of interactions with the environment to learn a reliablecontrol policy. This assumption of the availability of repeatedinteractions with the environment does not hold for many real-worldapplications due to safety concerns, costs/inconveniences related tointeractions, or the lack of an accurate simulator to enable effectivetraining in simulation prior to deployment and training in the realworld. Thus, practical application of reinforcement learning algorithmsin the real world is limited by its poor data efficiency and itsinflexibility of learning in an offline fashion.

In order to reduce the amount of time of interactions with theenvironment and to improve efficiency, model-based RL utilizes a learnedsystem model for predicting the system dynamics (i.e. states or rewards)and making a control plan accordingly. However, model-based methodssuffer from a model-bias problem, where certain model spaces areinaccurate, resulting in unstable policy learning.

SUMMARY

Example embodiments address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexample embodiments are not required to overcome the disadvantagesdescribed above, and may not overcome any of the problems describedabove.

According to an aspect of the disclosure, there is provided a server forobtaining a load balancing artificial intelligence (AI) model for aplurality of base stations in a communication system. The server mayinclude at least one memory storing instructions; and at least oneprocessor configured to execute the instructions to: obtain a pluralityof teacher models based on a plurality of traffic data sets collectedfrom the plurality of base stations, respectively; perform a policyrehearsal process including: obtaining a plurality of student modelsbased on knowledge distillation from the plurality of teacher models;obtaining an ensemble student model by ensembling the plurality ofstudent models; and obtaining a policy model by interacting with theensemble student model; provide the policy model to each of theplurality of base stations for a policy evaluation of the policy model;and based on a training continue signal being received from at least oneof the plurality of base stations as a result of the policy evaluation,update the ensemble student model and the policy model by performing thepolicy rehearsal process on the plurality of student models.

The least one processor may be further configured to execute theinstructions to: obtain the plurality of teacher models by receivingmodel parameters of the plurality of teacher models from the pluralityof base stations, and updating initialized model parameters of theplurality of teacher models based on the received model parameters.

The least one processor may be further configured to execute theinstructions to: obtain the plurality of teacher models by receiving theplurality of traffic data sets from the plurality of base stations, andtraining the plurality of teacher models based on the plurality oftraffic data sets, respectively.

The plurality of traffic data sets may include state-action-rewardtrajectories that include states, actions, and rewards. The states mayinclude at least one of an active user equipment (UE) number, abandwidth utilization, an internet protocol (IP) throughput, a cellphysical resource usage, and a speed of a download link. The actions mayinclude a load balancing parameter that causes the states to be changed.The rewards may include at least one of a minimum of IP throughput, atotal IP throughput, and a dead cell count.

Each of the plurality of teacher models may include a state transitionmodel and a reward transition model that are trained based onstate-action-reward trajectories that are collected from the pluralityof base stations. The state transition model may be configured to outputa predicted next state based on an action taken in a current state. Thereward transition model may be configured to output a predicted rewardbased on the action taken in the current state.

The obtaining the plurality of student models based on knowledgedistillation from the plurality of teacher models, may include:computing a ground-truth loss based on a difference between aground-truth value and a prediction of each of the plurality of studentmodels; computing a knowledge distillation loss based on a differencebetween a teacher prediction of the plurality of teacher models and astudent prediction of the plurality of student models; computing anaggregated loss that combines the ground-truth loss and the knowledgedistillation loss; and training the plurality of student models byminimizing or converging the aggregated loss.

The obtaining the policy model may include: obtaining state-reward pairsfrom the plurality of student models; computing an average of thestate-reward pairs; inputting the average of the state-reward pairs tothe policy model to obtain an action as an output of the policy model;increasing a time step by one; based on the increased time step beingless than a predetermined value, inputting the action to the pluralityof student models to continue the policy rehearsal process; and based onthe increased time step being equal to the predetermined value,terminating the policy rehearsal process and outputting the policymodel.

The training continue signal may indicate that a reward obtained fromthe ensemble student model is less than a reward obtained from anexisting load balancing model by a predetermined margin or more.

According to another aspect of the present disclosure, there is provideda method for obtaining a load balancing artificial intelligence (AI)model for a plurality of base stations in a communication system. Themethod may include: obtaining a plurality of teacher models based on aplurality of traffic data sets collected from the plurality of basestations, respectively; performing a policy rehearsal process by:obtaining a plurality of student models based on knowledge distillationfrom the plurality of teacher models; obtaining an ensemble studentmodel by ensembling the plurality of student models; and obtaining apolicy model by interacting with the ensemble student model;transmitting the policy model to each of the plurality of base stationsfor a policy evaluation of the policy model; and based on a trainingcontinue signal being received from at least one of the plurality ofbase stations as a result of the policy evaluation, updating theensemble student model and the policy model by performing the policyrehearsal process on the plurality of student models.

The obtaining of the plurality of teacher models may include: receivingmodel parameters of the plurality of teacher models from the pluralityof base stations; and updating initialized model parameters of theplurality of teacher models based on the received model parameters.

The obtaining of the plurality of teacher models may include: receivingthe plurality of traffic data sets from the plurality of base stations,and training the plurality of teacher models based on the plurality oftraffic data sets, respectively.

The plurality of traffic data sets may include state-action-rewardtrajectories. The state-action-reward trajectories may include states,actions, and rewards. The states may include at least one of an activeuser equipment (UE) number, a bandwidth utilization, an internetprotocol (IP) throughput, a cell physical resource usage, and a speed ofa download link. The actions may include a load balancing parameter thatcauses the states to be changed. The rewards may include at least one ofa minimum of IP throughput, a total IP throughput, and a dead cellcount.

Each of the plurality of teacher models may include a state transitionmodel and a reward transition model that are trained based onstate-action-reward trajectories that are collected from the pluralityof base stations. The state transition model may be configured to outputa predicted next state based on an action taken in a current state. Thereward transition model may be configured to output a predicted rewardbased on the action taken in the current state.

The obtaining the plurality of student models based on knowledgedistillation from the plurality of teacher models, may include:computing a ground-truth loss based on a difference between aground-truth value and a prediction of each of the plurality of studentmodels; computing a knowledge distillation loss based on a differencebetween a teacher prediction of the plurality of teacher models and astudent prediction of the plurality of student models; computing anaggregated loss that combines the ground-truth loss and the knowledgedistillation loss; and training the plurality of student models byminimizing or converging the aggregated loss.

The obtaining the policy model may include: obtaining state-reward pairsfrom the plurality of student models; computing an average of thestate-reward pairs; inputting the average of the state-reward pairs tothe policy model to obtain an action as an output of the policy model;increasing a time step by one; based on the increased time step beingless than a predetermined value, inputting the action to the pluralityof student models to continue the policy rehearsal process; and based onthe increased time step being equal to the predetermined value,terminating the policy rehearsal process and outputting the policymodel.

The training continue signal may indicate that a reward obtained fromthe ensemble student model is less than a reward obtained from anexisting load balancing model by a predetermined margin or more.

According to another aspect of the present disclosure, there is provideda non-transitory computer-readable storage medium storing a program thatis executable by at least one processor to perform the method forobtaining a load balancing artificial intelligence (AI) model for aplurality of base stations in a communication system.

Additional aspects will be set forth in part in the description thatfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and aspects of embodiments of thedisclosure will be more apparent from the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a diagram showing an overview of a system for performingtraffic load balancing according to embodiments of the presentdisclosure;

FIG. 2 is a diagram illustrating a method for generating a controlpolicy for performing traffic load balancing according to embodiments ofthe present disclosure;

FIGS. 3A and 3B are diagrams illustrating a structure of a teacher modelaccording to various embodiments of the present disclosure, and FIG. 3Cis a graph showing a relationship between a reward and the number ofteacher models that transfer knowledge to student models according toembodiments of the present disclosure;

FIG. 4 is a diagram illustrating a method of training student modelsaccording to embodiments of the present disclosure;

FIG. 5 is a diagram illustrating a method of combining student models toobtain an ensemble student model according to embodiments of the presentdisclosure;

FIG. 6 is a diagram illustrating a method of evaluating a policy modelaccording to embodiments of the present disclosure;

FIG. 7 is a flowchart illustrating a method of performing traffic loadbalancing according to embodiments of the present disclosure;

FIG. 8 is a flowchart illustrating another method of performing trafficload balancing according to embodiments of the present disclosure;

FIG. 9 is a flowchart illustrating a method of training teacher modelsaccording to embodiments of the present disclosure;

FIG. 10 is a flowchart illustrating a method of training student modelsand obtaining an ensemble student model according to embodiments of thepresent disclosure;

FIG. 11 is a flowchart illustrating a method of performing a policyrehearsal according to embodiments of the present disclosure;

FIG. 12 is a flowchart illustrating a method of performing a policyevaluation according to embodiments of the present disclosure;

FIG. 13 is a block diagram of an electronic device according toembodiments of the present disclosure;

FIG. 14 illustrates a use application in which a server allocates acommunication spectrum according to embodiments of the presentdisclosure;

FIG. 15 illustrates a use application in which a server performs trafficload balancing between different communication cells, according toembodiments of the present disclosure; and

FIG. 16 illustrates a cell reselection process according to embodimentsof the present disclosure.

DETAILED DESCRIPTION

Example embodiments are described in greater detail below with referenceto the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exampleembodiments. However, it is apparent that the example embodiments can bepracticed without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure the description with unnecessary detail.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list. For example, the expression, “at leastone of a, b, and c,” should be understood as including only a, only b,only c, both a and b, both a and c, both b and c, all of a, b, and c, orany variations of the aforementioned examples.

While such terms as “first,” “second,” etc., may be used to describevarious elements, such elements must not be limited to the above terms.The above terms may be used only to distinguish one element fromanother.

The term “component” is intended to be broadly construed as hardware,firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of possible implementations. In fact,many of these features may be combined in ways not specifically recitedin the claims and/or disclosed in the specification. Although eachdependent claim listed below may directly depend on only one claim, thedisclosure of possible implementations includes each dependent claim incombination with every other claim in the claim set.

No element, act, or instruction used herein should be construed ascritical or essential unless explicitly described as such. Also, as usedherein, the articles “a” and “an” are intended to include one or moreitems, and may be used interchangeably with “one or more.” Furthermore,as used herein, the term “set” is intended to include one or more items(e.g., related items, unrelated items, a combination of related andunrelated items, etc.), and may be used interchangeably with “one ormore.” Where only one item is intended, the term “one” or similarlanguage is used. Also, as used herein, the terms “has,” “have,”“having,” or the like are intended to be open-ended terms. Further, thephrase “based on” is intended to mean “based, at least in part, on”unless explicitly stated otherwise.

FIG. 1 is a diagram showing an overview of a system 100 for performingtraffic load balancing according to embodiments of the presentdisclosure. The system 100 may be used to balance communication trafficloads among a plurality of cells served by each of a plurality of basestations. However, the embodiments of the present disclosure are notlimited thereto, and the system 100 may be applied to any type of loadbalancing, for example, such as the balancing of electric loads,transportation traffic loads, and the like.

The system 100 may include an environment 110 and a server 120 thatcommunicates with the environment 110. The environment 110 may include acommunication system that provides a plurality of base stations and aplurality of (communication) cells managed by each of the plurality ofbase stations. The server 120 may obtain an observation result of thecommunication system to perform a multi-teacher model basedreinforcement learning (RL) algorithm (MOBA), which leverages aplurality of teacher artificial intelligence (AI) models (hereinafter,referred to as “teacher models”) to solve a model-bias problem. Theresult of observing the communication system may include trajectories ofstates, actions, and reward. The state-action-reward trajectories mayindicate a temporal sequence of states which have changed as a responseto actions taken in certain states, with rewards being received as aresult of taking each of the actions. In reinforcement learning, theterm “trajectory” may refer to a sequence of states and actions, or asequence of states, actions, and rewards. The states may include any oneor any combination of an active user equipment (UE) number, a bandwidthutilization, an internet protocol (IP) throughput, a cell physicalresource usage, and a speed of a download link. The actions may includea load balancing parameter that causes the states to be changed, and therewards may include any one or any combination of a minimum of IPthroughput, a total IP throughput, a dead cell count, and other systemmetrics.

In MOBA according to embodiments of the present disclosure, differentteacher models learn various instances of the communication system, andtransfer their learned knowledge to a plurality of student AI models(hereinafter, referred to as “student models) so that the student modelslearn a generalized dynamic model that covers a state space. In order toovercome the instability of multi-teacher knowledge transfer, the server120 may utilize the plurality of student models and apply an ensemblemethod to combine the plurality of student models. The server 120 maydetermine a control action for changing load balancing parameters of theplurality of base stations via an ensemble of the plurality of studentmodels.

According to embodiments of the disclosure, a teacher model and astudent model may include one or more neural networks, and modelparameters may refer to parameters of the one or more neural networks,for example, such as weights and biases applied to neurons, the numberof layers, the number of neurons in each layer, connections betweenlayers, connections between neurons, and the like.

FIG. 2 is a diagram illustrating a method 200 for generating a controlpolicy for performing traffic load balancing according to embodiments ofthe present disclosure.

The method 200 may include operation 210 of obtaining a plurality oftraffic datasets (e.g., Traffic Data #1, Traffic Data #2, . . . ,Traffic Data #N) collected from a plurality of base stations (e.g., BS#1, BS #2, . . . , BS #N), and storing the plurality of traffic datasetsin their corresponding replay buffers.

Each of the plurality of traffic datasets may include M data pointsβ={(s_(t), a_(t), r_(t), s′_(t))|t=1, . . . , M} to leverage MarkovDecision Process (MDP)-based reinforcement learning (RL), wherein sdenotes a current state, a denotes an action, r denotes a reward, and s′denotes a predicted next state when the action is taken in the currentstate. The term “action” may refer to a control action taken by thecommunication system or the base station to perform the traffic loadbalancing between multiple base stations or between multiple cellscovered by a single base station. For example, a control action ofadjusting threshold values for load balancing features may be set as the“action.” The term “reward” may refer to a value added to the currentstate in response to the “action” being taken at the current state. Forexample, a minimum IP throughput per cell may be set as the “reward” inembodiments of the present disclosure.

According to embodiments of the disclosure, the input of “state” may beexpressed as a combination of a first vector indicating an averagenumber of active user equipment (UEs) of each cell, a second vectorindicating an average bandwidth utilization value of each cell, and athird vector indicating an average throughput of each cell. When thereare four cells, a state may be expressed as [16.34, 15.25, 6.51, 2.91,0.85, 0.72, 0.59, 0.25, 1.29, 1.11, 1.54, 1.67], where “16.34, 15.25,6.51,” “2.91, 0.85, 0.72,” “0.59, 0.25, 1.29” and “1.11, 1.54, 1.67”correspond to the first, second, and third vectors for each of the fourcells. The input of “action” for adjusting load balancing parameters ofthe base stations may be expressed as, for example, [2.3, 3.6, 5.1, 0.5,1.0, 0.0, . . . , 5.5, 5.6, 3.1, 8.1, 9.9, 10.0] in a dB scale.

The method 200 may include operation 220 of obtaining a plurality ofteacher models (e.g., Teacher Model 1, Teacher Model 2, . . . , TeacherModel N), based on the traffic data collected from the plurality of basestations, respectively.

In operation 220, each of the plurality of teacher models may be trainedusing its own local traffic data, via a discrete-time finite Markovdecision process (MDP)-based RL in which a policy agent model aims tolearn an optimal control policy by interacting with the environment ofthe communication system. An RL problem may be formulated as a MarkovDecision Process (MDP), such as a tuple (S, A, p, r), wherein S denotesa state space, A denotes an action space, p: S⊗A→S′ denotes a statetransition function, r: S⊗A→R denotes a reward function. Each of theteacher models learns an agent policy configured to output a sequence ofstates and actions which can collect the largest expected return. Theexpected return may be expressed as η(θ)=

[Σ_(t) ^(T)γ^(t), r_(t)], where T denotes a preset time (e.g., 24hours), and γ a discount factor. At each iteration step, the teachermodels may update their model parameters to minimize a teacher loss andthereby to maximize a log-likelihood of a state transition distributionand a reward transition distribution. A loss is considered as beingminimized or converging when the loss has reached a preset minimumthreshold, or the loss does not reduce any longer and therefore hasreached a constant value (with a preset margin). The teacher loss may becomputed as expressed in Equation (1):

$\begin{matrix}{L_{T} = {\sum\limits_{k = 1}^{N}{\sum\limits_{{({s_{t},a_{t},s_{t + 1},r_{t}})} \in D_{k}}\lbrack {{{s_{t + 1} - {f_{\phi_{k}^{T}}( {s_{t},a_{t}} )}}}_{2}^{2} + {{r_{t} - {f_{\eta_{k}^{T}}( {s_{t},a_{t}} )}}}_{2}^{2}} \rbrack}}} & (1)\end{matrix}$

Where f_(ϕ) _(k) _(T) denotes the state transition model configured toreceive as inputs, a current state s_(t) and an action a_(t) to be takenin the current state s_(t), and output a predicted next state ŝ_(t+1),s_(t+1) denotes a ground-truth next state. f_(η) _(k) _(T) denotes thereward transition model configured to receive as inputs, the currentstate s_(t) and the action a_(t) to be taken in the current state s_(t),and output a predicted reward {circumflex over (r)}_(t+1) to be given asa result of taking the action a_(t) in the current state s_(t), andr_(t) is a ground-truth reward.

Operation 220 will be described in further detail with reference toFIGS. 3A-3C.

The method 200 may include operation 230 of obtaining a plurality ofstudent models (e.g., Student Model 1, Student Model 2, . . . , StudentModel K). The number of student models may be the same as or differentfrom the number of teacher models. Each student model may have the sameor substantially the same network structure as the teacher models. Forexample, each student model may include a state transition distributionand a reward transition distribution. The plurality of student modelsmay be initialized with different model parameters. For example,different sets of model parameters may be randomly and/or uniformlysampled from a plurality of sets of model parameters for theinitialization of the student models.

In operation 230, the plurality of teacher models (instead of trafficdata collected from base stations) may be aggregated via multi-teacherknowledge distillation to train a student model (e.g., Student Model 1)that provides an action for controlling its target base station (e.g.,BS #1). When there are plural student models, the multi-teacherknowledge is transferred from the plurality of teacher models (e.g.,Teacher Model 1, Teacher Model 2, . . . , Teacher Model N) to each ofthe student models (e.g. Student Model 1, Student Model 2, . . . ,Student Model K). A model aggregation according to an embodiment mayaddress a limited bandwidth issue of data aggregation.

In embodiments of the present disclosure, knowledge (e.g., teacherpredictions) distilled from the plurality of teacher models isintegrated and the integrated knowledge is provided to each of thestudent models to improve the prediction accuracy of each of the studentmodels. For example, an average of the predictions of the plurality ofteacher models may be provided to each of the student models as theintegrated knowledge. For these teacher models, each student model istrained via a knowledge distillation (KD) process to minimize orconverge a student loss that combines a ground-truth loss between aprediction of the student model and a ground-truth value, and a KD lossbetween the prediction of the student model and predictions of theteacher model. For example, the student loss may be expressed inEquation (2):L _(S)=Σ_(k=1) ^(N)Σ_((s) _(t) _(,a) _(t) _(,s) _(t+1) _()∈D) _(k) [∥s_(t+1) −f _(ϕ) _(s) (s _(t) ,a _(t))∥₂ ² +∥f _(ϕ) _(k) _(T) (s _(t) ,a_(t))−f _(ϕ) _(s) (s _(t) ,a _(t))∥₂ ²]  (2)

Where f_(ϕ) _(s) denote a student model, f_(ϕ) _(s) (s_(t), a_(t))denotes a predicted state of the student model, s_(t+1) denotes aground-truth state, and f_(ϕ) _(k) _(T) (s_(t), a_(t)) denotes apredicted state of the teacher models (e.g., an average of predictedstates of the teacher models). ∥s_(t+1)−f_(ϕ) _(s) (s_(t), a_(t))∥₂ ²represents the ground-truth loss, and ∥f_(ϕ) _(k) _(T) (s_(t),a_(t))−f_(ϕ) _(s) (s_(t), a_(t))∥₂ ² represents the KD loss.

Operation 230 will be described in further detail with reference to FIG.4 .

The method 200 may include operation 240 of obtaining an ensemble of theplurality of student models for a policy rehearsal. At each iterationtime step t, a student model computes a predicated state s_(t+1) andreward r_(t), which mirrors the structure of an MDP model that computesan approximate MDP model with an expected reward and state for a givenstate and action.

In operation 240, a state ensemble may be computed by averagingpredicted next states of the student models, and a reward ensemble maybe computed by averaging predicted rewards of the student models. Forexample, the state ensemble ŝ_(t+1) and the reward ensemble

may be expressed in Equations (3) and (4):

$\begin{matrix}{{\hat{s}}_{t + 1} = {\frac{1}{K}{\sum\limits_{k = 1}^{K}\lbrack {f_{\phi_{k}^{T}}( {s_{t},a_{t}} )} \rbrack}}} & (3)\end{matrix}$ $\begin{matrix}{{\hat{r}}_{t} = {\frac{1}{K}{\sum\limits_{k = 1}^{K}\lbrack {f_{\eta_{k}^{S}}( {s_{t},a_{t}} )} \rbrack}}} & (4)\end{matrix}$

where K is the total number of student models, f_(ϕ) _(s) is the statetransition model of the student model, and f_(η) _(s) is the rewardtransition model of the student model.

The state ensemble and the reward ensemble may be provided to an agentpolicy model (also referred to as “policy model”) which applies a policylearning algorithm, such as Proximal Policy Optimization (PPO), DeepDeterministic Policy Gradient (DDPG), Twin-delayed DDPG, or SoftActor-Critic (SAC), to learn and update a control policy. The agentpolicy model may be trained to minimize or converge a policy rehearsalloss, which decreases as the predicted return that is expressed inEquation (5) increases:{circumflex over (η)}(θ;ϕ_(S))=E _({circumflex over (τ)})[Σ^(T) _(t=0)r(s _(t) ,a _(t))]  (5)

The agent policy model may be trained to maximize the above-identifiedpredicted return, and thereby to minimize or converge the policyrehearsal loss.

Operation 240 will be described in further detail with reference to FIG.5 .

The method 200 may include operation 250 of evaluating policy actionsthat are provided from the ensemble of the plurality of student models,with interaction with the real communication environment.

In operation 250, a return is computed based on a new control policyapplied to the agent policy model. The agent policy model may output anaction to be taken in a current state based on the new control policy,and may collect a reward that is given as a result of taking the action.The expected return of the new control policy is computed by adding upthe collected rewards. For example, the expected return may be computedusing Equation (6):η(θ)=

[Σ^(T) _(t=0) r(s _(t) ,a _(t))]  (6)

Where

denotes an expectation function, and T denotes a predetermined number ofiteration time steps.

The return of the new control policy may be compared with a return of anold control policy. When the return of the new control policy is lessthan the return of the old control policy by a predetermined margin ormore, the new control policy is determined not to improve any longer,and therefore the policy learning is terminated. For example, the policylearning is terminated when the current control policy meets thefollowing Equation (7):

$\begin{matrix}{\frac{1}{T}{\sum\limits_{t = 1}^{T}\lbrack {{\eta( \theta_{new} )} < {{\eta( \theta_{old} )} + C}} \rbrack}} & (7)\end{matrix}$

Wherein

denotes an indicator function, which outputs a value 1 if the

equation hold, C denotes a predetermined margin, and T denotes apredetermined number of iteration time steps.

Operation 250 will be described in further detail with reference to FIG.6 .

FIGS. 3A and 3B are diagrams illustrating a structure of a teacher modelaccording to various embodiments of the present disclosure.

As shown in FIG. 3A, a teacher model may include an input layer, hiddenlayers, a first output layer configured to output a predicted state, anda second output layer configured to output a predicted reward. In orderto train the teacher model, a state transition model loss is computedbased on a difference between the predicted state and a ground-truthstate, and a reward transition model loss is computed based on adifference between the predicted reward and a ground-truth reward, andan overall loss that combines the state transition model loss and thereward transition model loss is back-propagated to update networkparameters of the hidden layers.

Referring to FIG. 3B, a teacher model may include an input layer, firsthidden layers, second hidden layers, a first output layer connected tothe first hidden layers and configured to output a predicted state, anda second output layer connected to the second hidden layers andconfigured to output a predicted reward. Unlike the network structurehaving shared hidden layers as illustrated in FIG. 3A, the networkstructure shown in FIG. 3B has two separate hidden layers for predictinga state and a reward, respectively. In order to train the teacher modelof FIG. 3B, a state transition model loss is computed based on adifference between the predicted state and a ground-truth state and thestate transition model loss is back-propagated to update networkparameters of the first hidden layers. Additionally, a reward transitionmodel loss is computed based on a difference between the predictedreward and a ground-truth reward, and the reward transition model lossis back-propagated to update network parameters of the second hiddenlayers. Although FIG. 3B illustrates that the input layer is shared withthe first hidden layers and the second hidden layers, the embodimentsare not limited thereto and two separate input layers may be provided.Also, student models according to embodiments of the present disclosuremay have the same or substantially the same network structure asillustrated in FIG. 3A or FIG. 3B.

FIG. 3C is a graph showing a relationship between a reward and thenumber of teacher models according to embodiments of the presentdisclosure.

As shown in FIG. 3 , a reward tends to decrease from a certain point asthe number of teacher models increases. Based on experiments, the numberof teacher models may be set to have a number in a range from four toeight. For example, six teacher models may be used in transferringknowledge to the student models to avoid the decrease in reward.

FIG. 4 is a diagram illustrating a method of training student modelsaccording to embodiments of the present disclosure.

As shown in FIG. 4 , the server 120 may utilize a plurality of teachermodels 1-N and a plurality of student models 1-K. The predictions of theplurality of teacher models 1-N may be integrated and then transferredto each of the plurality of student models 1-K. For example, an averagevalue of the predictions of the plurality of teacher models 1-N may beprovided to each of the plurality of student models 1-K.

Each of the plurality of student models 1-K may compute a student lossthat combines a distillation loss and a ground-truth loss. Thedistillation loss may represent a difference between a teacherprediction (e.g., the average value of the predictions of the pluralityof teacher models 1-N) and a student prediction of the student model.The ground-truth loss may represent a difference between the studentprediction and a ground-truth value.

When the teacher models 1-N and the student models 1-K are constitutedwith a state transition model and a reward transition model, the teacherprediction may include a teacher predicted state and a teacher predictedreward, and the student prediction may include a student predicted stateand a student predicted reward. The ground-truth value may include aground-truth state and a ground-truth reward. In that case, thedistillation loss may represent each or a combination of a differencebetween the teacher predicted state and the student predicted state, anda difference between the teacher predicted reward and the studentpredicted reward. The ground-truth loss may represent each or acombination of a difference between the student predicted state and theground-truth state and a difference between the student predicted rewardand the ground-truth reward.

In computing the distillation loss, any one or any combination of aKullback-Leibler (KL) divergence loss function, a negative loglikelihood loss function, and a mean squared error loss function may beused.

According to embodiments of the disclosure, the number of student modelsmay be determined to achieve a balance between a performance of anensemble student model and a computational cost caused by the number ofthe student models. The performance of the ensemble student modelincreases in proportion to the number of student models. However, whenthe number of the student models reaches a certain number, theperformance improvement becomes marginal, whereas the computational costcontinues to increase in proportion to the number of student models.Based on an evaluation with different numbers of student models, thenumber of student models may be set to have a number in a range from twoto six. For example, three student models may be used to obtain anensemble student model, but the embodiments are not limited thereto.

FIG. 5 is a diagram illustrating a method of combining student models toobtain an ensemble student model for a policy rehearsal according toembodiments of the present disclosure.

Referring to FIG. 5 , once the student models 1-K are trained inoperation 230 of FIG. 4 , a first intermediate state-reward pair, asecond intermediate state-reward pair, and a K^(th) intermediatestate-reward pair are obtained from the student models 1-K,respectively, in operation 240. In turn, an ensemble algorithm may beapplied to combine the first intermediate state-reward pair, the secondintermediate state-reward pair, and the K^(th) intermediate state-rewardpair. For example, an average of all intermediate state values, and anaverage of all intermediate reward values may be computed as a stateensemble and a reward ensemble, respectively. The state ensemble and thereward ensemble may input an agent policy model which applies a policylearning algorithm, such as Proximal Policy Optimization (PPO), DeepDeterministic Policy Gradient (DDPG), Twin-delayed DDPG, or SoftActor-Critic (SAC), to learn and update a control policy. The agentpolicy model may be trained to minimize or converge a policy rehearsalloss, which decreases as the predicted return expressed in Equation (5)increases.

The combination of the student models 1-K with the ensemble algorithmmay be considered as an ensemble student model.

FIG. 6 is a diagram illustrating a method of evaluating a policy modelaccording to embodiments of the present disclosure.

Referring to FIG. 6 , once the training of the agent policy model iscompleted via the policy rehearsal in operation 240 of FIG. 5 , theagent policy model may provide a control action (e.g., a control actionfor adjusting traffic load parameters of base stations) to the realenvironment including the base stations BS #1-BS #N and may obtain astate-reward pair (e.g., a communication system state indicting anaverage number of active UEs per cell, an average bandwidth utilizationper cell, an average IP throughput per cell, and a reward indicating aminimum IP throughput) via observation of the base stations BS #1-BS #N,in operation 250.

Based on the observation, the server 120 may determine whether the newcontrol policy applied to the agent policy model provides a higherperformance than an old control policy. For example, the server 120 maycompare a return of the new control policy with a return of the oldcontrol policy, and may determine the new control policy stops improvingwhen the return of the new control policy is less than the return of theold control policy by a predetermined margin or more. When the newcontrol policy is determined not to improve any longer, the server 120may stop the policy learning process.

FIG. 7 is a flowchart illustrating a method of performing traffic loadbalancing according to embodiments of the present disclosure.

In operation 701, a system including a server and a plurality of basestations are initiated.

In operation 702, the server initializes teacher models and studentmodels according to an existing load balancing model or an existingcontrol policy, so that the teacher models and the student models may beset up with an initialized set of model parameters.

In operations 703 and 705, each base station may collect its own localtraffic dataset, sample state-action-reward trajectories from thetraffic data set, add the sampled state-action-reward trajectories toits local relay buffer, and train a teacher model using thestate-action-reward trajectories. Operations 703 and 705 may correspondto operations 210 and 220 illustrated in FIG. 2 .

In operations 704 and 706, when each of the base stations finishestraining its teacher model, each of the base stations may transmit modelparameters of the teacher model to the server.

In operation 707, the server may update the initialized teacher modelsbased on the teacher model parameters transmitted from the basestations, and perform a teacher model interface to obtain teacher'spredicted state-reward pairs as outputs of the teacher models.

In operation 708, the server may train the student models based on theteacher's predicted state-reward pairs and the state-action-rewardtrajectories provided from each of the base stations. For example, theserver may compute a distillation loss that represents a differencebetween a prediction of the teacher models and a prediction of each ofthe student models, and a ground-truth loss that represents a differencebetween the prediction of each of the student models and a ground-truthvalue, and may train each of the student models to minimize or convergea sum of the distillation loss and the ground-truth loss. The server mayuse Equation (2) to compute the distillation loss and the ground-truthvalue. Operation 708 may correspond to operation 230 illustrated inFIGS. 2 and 4 .

In operation 709, the server may perform a policy rehearsal on anensemble of the student models. The ensemble of the student models maybe obtained by computing an average of predicted states of the studentmodels as a state ensemble, computing an average of predicted rewards ofthe student models as a reward ensemble, and providing the stateensemble and the reward ensemble rewards to an agent policy model toobtain an updated state ensemble and an update reward ensemble via aniteration process. For example, the server may use Equations (3) and (4)to compute the state ensemble and the reward ensemble, respectively, andperform the iteration process until a predicted reward of the agentpolicy model is maximized, for example using Equation (5). Operation 709may correspond to operation 240 illustrated in FIGS. 2 and 5 .

In operation 710, the server may perform a policy evaluation todetermine whether a new control policy applied by the ensemble studentmodel to an agent policy model continues to improve, in comparison withthe performance of an old control policy. When a return of the newcontrol policy is less than a return of the old control policy by apredetermined marine or more, the new control policy is determined notto improve any longer and therefore the policy learning is terminated.Operation 710 may correspond to operation 250 illustrated in FIGS. 2 and6 .

In operations 711 and 712, after the policy learning is completed, theserver may transmit the new control policy to each of the base stations.

In operations 713 and 714, each of the base stations may perform atraffic load balancing operation based on the new control policy.

FIG. 8 is a flowchart illustrating another method of performing trafficload balancing according to embodiments of the present disclosure.

Operations 801 and 807-813 may be performed in the same or substantiallythe same manner as operations 701 and 708-714, and therefore duplicatedescription will be omitted for conciseness.

In operation 802 and 804, each base station may not train its ownteacher model, and instead, may transmit the state-action-rewardtrajectories that are sampled from its replay buffer to the server, inoperations 803 and 805.

In operation 806, the server may train the teacher models based on thestate-action-reward trajectories received from each of the basestations, so as to transfer knowledge of the teacher models to thestudent models.

As such, the training of the teacher models may be performed in each ofthe base stations as shown in FIG. 7 , or alternatively, may beperformed in the server as shown in FIG. 8 .

FIG. 9 is a flowchart illustrating a method of training teacher modelsaccording to embodiments of the present disclosure. FIG. 9 illustrates amethod of training a single teacher model, but the method may be appliedto each of a plurality of teacher models in the same or substantiallythe same manner.

In operation 901, state-action-reward trajectories that are sampled froma replay buffer may be input to a teacher model.

In operation 902, the teacher model may be trained to minimize orconverge a teacher loss. The teacher loss may include a state transitionmodel loss representing a difference between a predicted next state ofthe teacher model and a ground-truth next state, and a reward transitionmodel loss representing a difference between a predicted reward of theteacher model and a ground-truth reward. The teacher loss, the statetransition model loss, and the reward transition model loss may becomputed using Equation (1).

In operation 903, a state transition model of the teacher model isobtained by minimizing or converging the state transition model loss orthe teacher loss.

In operation 904, a reward transition model of the teacher model isobtained by minimizing or converging the reward transition model loss orthe teacher loss.

FIG. 10 is a flowchart illustrating a method of training student modelsand obtaining an ensemble student model according to embodiments of thepresent disclosure.

In operation 1001, state-action-reward trajectories (s_(t), a_(t),r_(t)) that are sampled from a replay buffer, may be input to a studentmodel.

In operation 1002, teacher predicted states (s_(t) ¹, s_(t) ² . . . ,s_(t) ^(N)) that are output from each of the state transition models ofthe teacher models 1-N, may be input to the student model.

In operation 1003, teacher predicted rewards (r_(t) ¹, r_(t) ² . . . ,r_(t) ^(N)) that are output from each of the reward transition models ofthe teacher models 1-N, may be input to the student model.

In operation 1004, a state transition model of the student model may betrained using the state-action pairs (s_(t), a_(t)) sampled from thereplay buffer and the teacher predicted states (s_(t) ¹, s_(t) ² . . . ,s_(t) ^(N)) until a state transition model loss of the student model isminimized or converges. The state transition model loss may be computedusing Equation (2).

In operation 1005, a reward transition model of the student model may betrained using the reward (r_(t)) sampled from the replay buffer and theteacher predicted rewards (r_(t) ¹, r_(t) ² . . . , r_(t) ^(N)) until areward transition model loss of the student model is minimized orconverges. The reward transition model loss may be computed usingEquation (2).

Each of a plurality of student models may be trained via operations1001-1005. Operations 1001-1005 may correspond to operation 230illustrated in FIGS. 2 and 3 .

In operation 1006, intermediate states are obtained from the statetransition models of the plurality of student models.

In operation 1007, intermediate rewards are obtained from the rewardtransition models of the plurality of student models.

In operation 1008, a state ensemble may be obtained by averaging theintermediate states, and a reward ensemble may be obtained by averagingthe intermediate rewards.

FIG. 11 is a flowchart illustrating a method of performing a policyrehearsal according to embodiments of the present disclosure.

The method of performing a policy rehearsal may include operations1101-1107.

In operations 1101 and 1102, a plurality of student models 1-K areobtained via knowledge distillation from a plurality of teacher models.

In operation 1102, intermediate state-reward pairs (ŝ_(t) ¹ and{circumflex over (r)}_(t) ¹, ŝ_(t) ² and {circumflex over (r)}_(t) ², .. . , and ŝ_(t) ^(N) and {circumflex over (r)}_(t) ^(N)) are obtainedfrom the outputs of the plurality of student models 1-K.

In operation 1103, all the intermediate states are combined as an stateensemble ŝ_(t), and all the intermediate rewards are combined as areward ensemble {circumflex over (r)}_(t). The state ensemble ŝ_(t) andthe reward ensemble {circumflex over (r)}_(t) may be computed usingEquations (3) and (4).

In operation 1104, an agent policy model may be trained using the stateensemble ŝ_(t) and the reward ensemble {circumflex over (r)}_(t), tomaximize a predicted return via a policy gradient method. At eachiteration time step, policy parameters may be updated as follows:

$\begin{matrix}{\theta_{k + 1} = {\arg\max\limits_{\theta}\frac{1}{{❘D_{k}❘}T}{\sum\limits_{\tau \in D_{k}}{\sum\limits_{t = 0}^{T}{\min( {{\frac{\pi_{\theta}( a_{t} \middle| s_{t} )}{\pi_{\theta_{k}}( a_{t} \middle| s_{t} )}{A^{\pi_{\theta_{k}}}( {s_{t},a_{t}} )}},{g( {\epsilon,{A^{\pi_{\theta_{k}}}( {s_{t},a_{t}} )}} )}} )}}}}} & (8)\end{matrix}$

Where θ_(k+1) denotes updated parameters at iteration time step k+1, kdenotes an iteration time step, π_(θk) denotes a policy parameterized byparameters θ_(k), and π_(θk+1) denotes a policy parameterized byparameters θ_(k+1). In other words, π_(k+1) represents a new controlpolicy that is updated from the current control policy π_(θk). “min”denotes a minimum function which chooses the lowest value among thecomponents of the minimum function, and “A” denotes an advantagefunction, which is expressed as A^(π)(s_(t), a_(t))=Q^(π)(s_(t),a_(t))−V^(π)(s_(t)), wherein Q^(π)(s_(t), a_(t)) refers to anactive-value function that shows an expected return when an action a istake in a certain state s, and V^(π)(s_(t)) refers to a state-valuefunction that shows an expected return for selecting a certain state s.g (ϵ, A) may be expressed as Equation (9):

$\begin{matrix}{{g( {\epsilon,A} )} = \begin{matrix}{( {1 + \epsilon} )A} & {A \geq 0} \\{( {1 - \epsilon} )A} & {A < 0}\end{matrix}} & (9)\end{matrix}$

After the training process of the agent policy model, an iteration timestep t is increased by 1 in operation 1105, and it is determined whetherthe increased iteration time t is less than a predetermined number ofiteration time steps T in operation 1106.

In operation 1106, when the increased iteration time t is less than thepredetermined number of iteration time steps T, a control action a_(t)that is output from the agent policy model is provided to each of thestudent models 1-K to repeat operations 1101-1106 until the iterationtime step t reaches the predetermined number of iteration time steps T.

When the iteration time step t teaches the predetermined number ofiteration time steps T, the policy rehearsal is terminated and the agentpolicy model is output, in operation 1107.

Operations 1011-1107 may correspond to operation 240 illustrated inFIGS. 2 and 5 .

FIG. 12 is a flowchart illustrating a method of performing a policyevaluation according to embodiments of the present disclosure.

The method of performing a policy evaluation may include operations1201-1210.

In operation 1201, a server may input an agent policy model that istrained via operations 240 illustrated in FIG. 2 or operations 1011-1107illustrated in FIG. 11 .

In operations 1201 and 1203, the server may transmit model parameters ofthe agent policy model to each of a plurality of base stations.

In operations 1204 and 1205, each of the plurality of base stations mayevaluate a new control policy provided from the agent policy model, incomparison with an old control policy.

In operations 1206 and 1207, each base station determines whether areturn of the new control policy is less than a return of the oldcontrol policy by a predetermined margin C or more. If the return of thenew control policy is less than the return of the old control policy bythe predetermined margin C or more, the base station(s) transmitsfeedback information such as a training continue signal, and otherwise,sends a training stop signal or does not send any signal. The feedbackinformation may provide information about an evaluation result of thenew control policy in comparison with the old control policy, forexample, information about whether the return of the new control policyis less than the return of the old control policy by the predeterminedmargin C or more.

In operation 1208, when the server receives a training continue signalfrom any of the base stations, the server performs a policy rehearsalprocess in operation 1209. When the server receives a training stopsignal or alternatively, does not receive a training continue signal,the server stops the policy rehearsal process in operation 1210.

Operations 1201-1209 may correspond to operation 250 illustrated inFIGS. 2 and 6 .

FIG. 13 is a block diagram of an electronic device 1300 according toembodiments.

FIG. 13 is for illustration only, and other embodiments of theelectronic device 1300 could be used without departing from the scope ofthis disclosure. For example, the electronic device 1300 may correspondto the server 120.

The electronic device 1300 includes a bus 1010, a processor 1320, amemory 1330, an interface 1340, and a display 1350.

The bus 1010 includes a circuit for connecting the components 1320 to1350 with one another. The bus 1010 functions as a communication systemfor transferring data between the components 1320 to 1350 or betweenelectronic devices.

The processor 1320 includes one or more of a central processing unit(CPU), a graphics processor unit (GPU), an accelerated processing unit(APU), a many integrated core (MIC), a field-programmable gate array(FPGA), or a digital signal processor (DSP). The processor 1320 is ableto perform control of any one or any combination of the other componentsof the electronic device 1300, and/or perform an operation or dataprocessing relating to communication. For example, the processor 1320performs operations 210-250 illustrated in FIG. 2 , and operations 702and 707-712 illustrated in FIG. 7 , operations 901-904 illustrated inFIG. 9 , operations 1001-1008 illustrated in FIG. 10 , operations1101-1107 illustrated in FIG. 11 , and operations 1201-1203 and1208-1210 illustrated in FIG. 12 . The processor 1320 executes one ormore programs stored in the memory 1330.

The memory 1330 may include a volatile and/or non-volatile memory. Thememory 1330 stores information, such as one or more of commands, data,programs (one or more instructions), applications 1334, etc., which arerelated to at least one other component of the electronic device 1300and for driving and controlling the electronic device 1300. For example,commands and/or data may formulate an operating system (OS) 1332.Information stored in the memory 1330 may be executed by the processor1320.

In particular, the memory 1330 stores data, computer-readableinstructions, applications, and setting information for the operation ofbase stations of the communication system 110. The memory 1330 may storeinformation on a bearer allocated to an accessed UE and a measurementresult reported from the accessed UE.

The applications 1334 include the above-discussed embodiments. Thesefunctions can be performed by a single application or by multipleapplications that each carry out one or more of these functions. Forexample, the applications 1334 may include artificial intelligence (AI)models for performing operations 210-250 illustrated in FIG. 2 , andoperations 702 and 707-712 illustrated in FIG. 7 , operations 901-904illustrated in FIG. 9 , operations 1001-1008 illustrated in FIG. 10 ,operations 1101-1107 illustrated in FIG. 11 , and operations 1201-1203and 1208-1210 illustrated in FIG. 12 . Specifically, the applications1334 may include teacher models 1334, student models 1336, and an agentpolicy model 1337 according to embodiments of the disclosure.

The display 1350 includes, for example, a liquid crystal display (LCD),a light emitting diode (LED) display, an organic light emitting diode(OLED) display, a quantum-dot light emitting diode (QLED) display, amicroelectromechanical systems (MEMS) display, or an electronic paperdisplay.

The interface 1340 includes input/output (I/O) interface 1342,communication interface 1344, and/or one or more sensors 1346. The I/Ointerface 1342 serves as an interface that can, for example, transfercommands and/or data between a user and/or other external devices andother component(s) of the electronic device 1300.

The communication interface 1344 may include a transceiver 1345 toenable communication between the electronic device 1300 and otherexternal devices (e.g., a plurality of base stations, and other serversthat may store teacher models), via a wired connection, a wirelessconnection, or a combination of wired and wireless connections. Thecommunication interface 1344 may permit the electronic device 1300 toreceive information from another device and/or provide information toanother device. For example, the communication interface 1344 mayinclude an Ethernet interface, an optical interface, a coaxialinterface, an infrared interface, a radio frequency (RF) interface, auniversal serial bus (USB) interface, a Wi-Fi interface, a cellularnetwork interface, or the like.

The transceiver 1345 of the communication interface 1344 may include aradio frequency (RF) circuitry 1345A and a baseband circuitry 1345B.

The baseband circuitry 1345B may transmit and receive a signal through awireless channel, and may perform band conversion and amplification onthe signal. The RF circuitry 1345A may up-convert a baseband signalprovided from the baseband circuitry 1345B into an RF band signal andthen transmits the converted signal through an antenna, anddown-converts an RF band signal received through the antenna into abaseband signal. For example, the RF circuitry 1345A may include atransmission filter, a reception filter, an amplifier, a mixer, anoscillator, a digital-to-analog converter (DAC), and ananalog-to-digital converter (ADC).

The transceiver 1345 may be connected to one or more antennas. The RFcircuitry 1345A of the transceiver 1345 may include a plurality of RFchains and may perform beamforming. For the beamforming, the RFcircuitry 1345A may control a phase and a size of each of the signalstransmitted and received through a plurality of antennas or antennaelements. The RF circuitry 1345A may perform a downlink multi-input andmulti-output (MIMO) operation by transmitting one or more layers.

The baseband circuitry 1345A may perform conversion between a basebandsignal and a bitstream according to a physical layer standard of theradio access technology. For example, when data is transmitted, thebaseband circuitry 1345B generates complex symbols by encoding andmodulating a transmission bitstream. When data is received, the basebandcircuitry 1345B reconstructs a reception bitstream by demodulating anddecoding a baseband signal provided from the RF circuitry 1345A.

The sensor(s) 1346 of the interface 1340 can meter a physical quantityor detect an activation state of the electronic device 1300 and convertmetered or detected information into an electrical signal. For example,the sensor(s) 1346 can include one or more cameras or other imagingsensors for capturing images of scenes. The sensor(s) 1346 can alsoinclude any one or any combination of a microphone, a keyboard, a mouse,and one or more buttons for touch input. The sensor(s) 1346 can furtherinclude an inertial measurement unit. In addition, the sensor(s) 1346can include a control circuit for controlling at least one of thesensors included herein. Any of these sensor(s) 1346 can be locatedwithin or coupled to the electronic device 1300.

Referring back to the processor 1320, the processor 1320 may transmitand receive signals through the RF circuitry 1345A and the basebandcircuitry 1345B. The processor 1320 may record data (e.g., traffic dataand/or model parameters) in the memory 1330 and read the data from thememory 1330.

For example, when the electronic device 1300 corresponds to the server120, the processor 1320 may receive from a communication system 110,traffic data, such as information about a number of active UEs that areserved by each cell of the base stations, a cell load ratio, and aninternet protocol (IP) throughput per cell, and may store theinformation of the number of active UEs, the cell load ratio, and the PIthroughput per cell, in the memory 1330. The processor 1320 may controlthe transceiver 1345 to transmit a request for traffic data to thecommunication system 110, and to receive from the server 120 theinformation of the number of active UEs, the cell load ratio, and the IPthroughput per cell, in response to the request from the traffic data.The processor 1320 may perform operations 210-250 based on thecommunication system state information, and may transmit a controlaction for adjusting load balancing parameters of the base stations tothe communication system 110. The communication system 110 may allocatecommunication bandwidth or UEs to the plurality of base stations of thecommunication system 110 or to the plurality of cells that are served byeach of the base stations, according to a control action received fromthe server 120, so that traffic loads are distributed relatively evenlyamong the plurality of base stations, and/or among the plurality ofcells of each base station.

FIG. 14 illustrates a use application in which a server performs trafficload balancing between different communication cells, according toembodiments.

Referring to FIG. 14 , a system for performing traffic load balancingaccording to an example embodiment includes a server 120, a plurality ofbase stations BS1-BS7 each of which serves a plurality of cells havingdifferent cell reselection priorities, and a plurality of UEs that arerespectively served in the plurality of cells.

In an example embodiment, a base station BS1 may serve a plurality ofcells C₁-C₇ having different frequency bands f₁-f₇ and different cellreselection priorities.

The server 120 may communicate with the plurality of base stationsBS1-BS7 to receive information about the state of the UEs in theirserving cells, for example, whether the UEs are in an idle mode or anactive mode, the number of active UEs, and an internet protocol (IP)throughput of each cell.

The server 120 may determine a cell reselection priority for each of theplurality of cells C₁-C₇ of the base station BS1 based on a controlaction provided from the server 120 via operations 210-250. For example,the server 120 may transmit a control action that adjusts the cellreselection priorities and/or the minimum IP throughput for each cell,to the base station BS1. Based on the control action, the base stationBS1 may reassign some of the plurality of UEs to another cell todistribute traffic load among the plurality of cells C1-C7.

FIG. 15 illustrates a cell reselection process according to an exampleembodiment.

As shown in FIG. 15 , a communication system includes at least one basestation (BS), a communication network, and a plurality of user equipment(UEs) that access the communication network through the at least one BS.

The at least one BS may correspond to an Evolved Node B (eNB), a NextGeneration Node B (gNB), a 6G Node. The BS may collect statusinformation of the UEs and may provide the UEs with access to thecommunication network based on the status information. Examples of thestatus information may include information of whether the UEs are in anactive mode or an idle mode, and may also include a buffer status, anavailable transmission power status, and a channel status of each of theUEs.

The communication system provides a first cell Cell 1 and a second cellCell 2, that are served by a base station BS1. For example, when six (6)UEs are connected to Cell 1 and one (1) cell is connected to Cell 2, oneor more UEs among the six UEs in Cell 2 are reassigned to Cell 1 todistribute communication traffic load between Cell 1 and Cell 2,according to a control action provided from the server.

Specifically, in an LTE, a 5G system, or a 6G system, the base stationBS1 may determine a cell reselection priority for each cell Cell 1 andCell 2 to which the UEs should connect, through a radio resource controlreleasing message. The UEs may determine a target cell on which to campbased on the cell reselection priority. For each UE, the cellreselection process is performed as a probabilistic process based on thecell reselection priority. When Cell 1 has a high cell reselectionpriority, a given idle mode UE may have a high probability of beingreselected to camp on Cell 1. The communication system may shift idleUEs from overloaded Cell 2 to less loaded Cell 1.

FIG. 16 illustrates a method of communicating with a UE and a BS toperform a cell reselection process according to an example embodiment.

As shown in FIG. 16 , the UE 121 in an idle mode may perform an initialcell selection in operation 1601. In order to select an initial cell,the UE 121 may scan all radio frequency (RF) channels in its operatingfrequency bands and may select an initial cell for the UE to camp on,based on cell selection criterion. For example, the UE 121 may selectthe initial cell based on various parameters, such as for example, acell selection reception (RX) level value (Srxlev), a cell selectionquality value (Squal), an offset temporarily applied to a cell(Qoffsettemp), a measured cell reception level value (Qqualmeas), ameasured cell quality value (Qrxlevmeas), a minimum required RX level inthe cell (Qrxlevmin), a minimum required quality level in the cell(Qqualmin). The UE 121 transmits information of the selected initialcell to a base station 122 that manages a plurality of cells, so thatthe UE 121 in the idle mode camps on the selected initial cell among theplurality of cells.

In operation 1602, the base station 122 may transmit traffic data,including the number of active mode UEs per cell, the cell load ratio,and the IP throughput per cell, to the server 120.

In operation 1603, the server 120 may determine cell reselectionparameters based on a new control policy that is generated viaoperations 210-250, and may transmit the cell reselection parameters tothe base station 122. The cell reselection parameters may correspond tocell reselection priorities that are assigned to the plurality of cellsC₁-C₇ shown in FIG. 14 .

In operation 1604, the base station 122 may transmit a Radio ResourceControl (RRC) Release message including the cell reselection parameters,to the UE 121.

In operation 1605, the UE 121 then may select a target cell to camp onbased on the cell reselection parameters, and may send information ofthe selected target cell to the base station 122. For example, when asecond cell C₂ has a higher cell reselection priority than the otherneighboring cells, C₁ and C₃-C₇, among the plurality of cells C₁-C₇, theidle mode UE 121 has a higher probability of being reassigned to camp onthe second cell C₂ than other neighboring cells, C₁ and C₃-C₇.

The method of generating a control policy and performing traffic loadbalancing according to the control policy may be written ascomputer-executable programs or instructions that may be stored in amedium.

The medium may continuously store the computer-executable programs orinstructions, or temporarily store the computer-executable programs orinstructions for execution or downloading. Also, the medium may be anyone of various recording media or storage media in which a single pieceor plurality of pieces of hardware are combined, and the medium is notlimited to a medium directly connected to electronic device 100, but maybe distributed on a network. Examples of the medium include magneticmedia, such as a hard disk, a floppy disk, and a magnetic tape, opticalrecording media, such as CD-ROM and DVD, magneto-optical media such as afloptical disk, and ROM, RAM, and a flash memory, which are configuredto store program instructions. Other examples of the medium includerecording media and storage media managed by application storesdistributing applications or by websites, servers, and the likesupplying or distributing other various types of software.

The forecasting method may be provided in a form of downloadablesoftware. A computer program product may include a product (for example,a downloadable application) in a form of a software programelectronically distributed through a manufacturer or an electronicmarket. For electronic distribution, at least a part of the softwareprogram may be stored in a storage medium or may be temporarilygenerated. In this case, the storage medium may be a server or a storagemedium of the server.

The foregoing disclosure provides illustration and description, but isnot intended to be exhaustive or to limit the implementation to theprecise form disclosed. Modifications and variations are possible inlight of the above disclosure or may be acquired from practice of theimplementation.

As used herein, the term “component” is intended to be broadly construedas hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of possible implementations. In fact,many of these features may be combined in ways not specifically recitedin the claims and/or disclosed in the specification. Although eachdependent claim listed below may directly depend on only one claim, thedisclosure of possible implementations includes each dependent claim incombination with every other claim in the claim set.

The embodiments of the disclosure described above may be written ascomputer executable programs or instructions that may be stored in amedium.

The medium may continuously store the computer-executable programs orinstructions, or temporarily store the computer-executable programs orinstructions for execution or downloading. Also, the medium may be anyone of various recording media or storage media in which a single pieceor plurality of pieces of hardware are combined, and the medium is notlimited to a medium directly connected to electronic device 1300, butmay be distributed on a network. Examples of the medium include magneticmedia, such as a hard disk, a floppy disk, and a magnetic tape, opticalrecording media, such as CD-ROM and DVD, magneto-optical media such as afloptical disk, and ROM, RAM, and a flash memory, which are configuredto store program instructions. Other examples of the medium includerecording media and storage media managed by application storesdistributing applications or by websites, servers, and the likesupplying or distributing other various types of software.

The above described method may be provided in a form of downloadablesoftware. A computer program product may include a product (for example,a downloadable application) in a form of a software programelectronically distributed through a manufacturer or an electronicmarket. For electronic distribution, at least a part of the softwareprogram may be stored in a storage medium or may be temporarilygenerated. In this case, the storage medium may be a server or a storagemedium of the electronic device 1300.

A model related to the neural networks described above may beimplemented via a software module. When the model is implemented via asoftware module (for example, a program module including instructions),the model may be stored in a computer-readable recording medium.

Also, the model may be a part of the electronic device 1300 describedabove by being integrated in a form of a hardware chip. For example, themodel may be manufactured in a form of a dedicated hardware chip forartificial intelligence, or may be manufactured as a part of an existinggeneral-purpose processor (for example, a CPU or application processor)or a graphic-dedicated processor (for example a GPU).

Also, the model may be provided in a form of downloadable software. Acomputer program product may include a product (for example, adownloadable application) in a form of a software program electronicallydistributed through a manufacturer or an electronic market. Forelectronic distribution, at least a part of the software program may bestored in a storage medium or may be temporarily generated. In thiscase, the storage medium may be a server of the manufacturer orelectronic market, or a storage medium of a relay server.

While the embodiments of the disclosure have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope as defined by thefollowing claims.

What is claimed is:
 1. A server for obtaining a load balancingartificial intelligence (AI) model for a plurality of base stations in acommunication system, the server comprising: at least one memory storinginstructions; and at least one processor configured to execute theinstructions to: obtain a plurality of teacher models based on aplurality of traffic data sets collected from the plurality of basestations, respectively; obtain a plurality of student models based onknowledge distillation from the plurality of teacher models; obtain anensemble student model by ensembling the plurality of student models;and transmit the ensemble student model to the plurality of basestations, respectively; receive feedback information of the ensemblestudent model from the plurality of base stations, and update theensemble student model based on the received feedback information. 2.The server of claim 1, wherein the least one processor is furtherconfigured to execute the instructions to: obtain the plurality ofteacher models by receiving model parameters of the plurality of teachermodels from the plurality of base stations, and updating initializedmodel parameters of the plurality of teacher models based on thereceived model parameters.
 3. The server of claim 1, wherein the leastone processor is further configured to execute the instructions to:obtain the plurality of teacher models by receiving the plurality oftraffic data sets from the plurality of base stations, and training theplurality of teacher models based on the plurality of traffic data sets,respectively.
 4. The server of claim 1, wherein the plurality of trafficdata sets comprise state-action-reward trajectories that comprisestates, actions, and rewards, the states comprise at least one of anactive user equipment (UE) number, a bandwidth utilization, an internetprotocol (IP) throughput, a cell physical resource usage, and a speed ofa download link, the actions comprise a load balancing parameter thatcauses the states to be changed, and the rewards comprise at least oneof a minimum of IP throughput, a total IP throughput, and a dead cellcount.
 5. The server of claim 1, wherein each of the plurality ofteacher models comprises a state transition model and a rewardtransition model that are trained based on state-action-rewardtrajectories that are collected from the plurality of base stations,wherein the state transition model is configured to output a predictednext state based on an action taken in a current state, and wherein thereward transition model is configured to output a predicted reward basedon the action taken in the current state.
 6. The server of claim 1,wherein the obtaining the plurality of student models based on knowledgedistillation from the plurality of teacher models, comprises: computinga ground-truth loss based on a difference between a ground-truth valueand a prediction of each of the plurality of student models; computing aknowledge distillation loss based on a difference between a teacherprediction of the plurality of teacher models and a student predictionof the plurality of student models; computing an aggregated loss thatcombines the ground-truth loss and the knowledge distillation loss; andtraining the plurality of student models by minimizing or converging theaggregated loss.
 7. The server of claim 1, wherein the least oneprocessor is further configured to execute the instructions to obtain apolicy model by: obtaining state-reward pairs from the plurality ofstudent models; computing an average of the state-reward pairs;inputting the average of the state-reward pairs to the policy model toobtain an action as an output of the policy model; increasing a timestep by one; based on the increased time step being less than apredetermined value, inputting the action to the plurality of studentmodels; and based on the increased time step being equal to thepredetermined value, outputting the policy model.
 8. The server of claim1, wherein the least one processor is further configured to execute theinstructions to: obtain a policy model by interacting with the ensemblestudent model; provide the policy model to each of the plurality of basestations for a policy evaluation of the policy model; and based on atraining continue signal being received from at least one of theplurality of base stations as a result of the policy evaluation, updatethe ensemble student model and the policy model, wherein the trainingcontinue signal is provided as the feedback information and indicatesthat a reward obtained from the ensemble student model is less than areward obtained from an existing load balancing model by a predeterminedmargin or more.
 9. A method for obtaining a load balancing artificialintelligence (AI) model for a plurality of base stations in acommunication system, the method comprising: obtaining a plurality ofteacher models based on a plurality of traffic data sets collected fromthe plurality of base stations, respectively; obtaining a plurality ofstudent models based on knowledge distillation from the plurality ofteacher models; obtaining an ensemble student model by ensembling theplurality of student models; and transmitting the ensemble student modelto the plurality of base stations, respectively; receiving feedbackinformation of the ensemble student model from the plurality of basestations, and updating the ensemble student model based on the receivedfeedback information.
 10. The method of claim 9, wherein the obtainingof the plurality of teacher models comprises: receiving model parametersof the plurality of teacher models from the plurality of base stations;and updating initialized model parameters of the plurality of teachermodels based on the received model parameters.
 11. The method of claim9, wherein the obtaining of the plurality of teacher models comprises:receiving the plurality of traffic data sets from the plurality of basestations, and training the plurality of teacher models based on theplurality of traffic data sets, respectively.
 12. The method of claim 9,wherein the plurality of traffic data sets comprise state-action-rewardtrajectories that comprise states, actions, and rewards, the statescomprise at least one of an active user equipment (UE) number, abandwidth utilization, an internet protocol (IP) throughput, a cellphysical resource usage, and a speed of a download link, the actionscomprise a load balancing parameter that causes the states to bechanged, and the rewards comprise at least one of a minimum of IPthroughput, a total IP throughput, and a dead cell count.
 13. The methodof claim 9, wherein each of the plurality of teacher models comprises astate transition model and a reward transition model that are trainedbased on state-action-reward trajectories that are collected from theplurality of base stations, wherein the state transition model isconfigured to output a predicted next state based on an action taken ina current state, and wherein the reward transition model is configuredto output a predicted reward based on the action taken in the currentstate.
 14. The method of claim 9, wherein the obtaining the plurality ofstudent models based on knowledge distillation from the plurality ofteacher models, comprises: computing a ground-truth loss based on adifference between a ground-truth value and a prediction of each of theplurality of student models; computing a knowledge distillation lossbased on a difference between a teacher prediction of the plurality ofteacher models and a student prediction of the plurality of studentmodels; computing an aggregated loss that combines the ground-truth lossand the knowledge distillation loss; and training the plurality ofstudent models by minimizing or converging the aggregated loss.
 15. Themethod of claim 9, further comprising obtaining a policy model by:obtaining state-reward pairs from the plurality of student models;computing an average of the state-reward pairs; inputting the average ofthe state-reward pairs to the policy model to obtain an action as anoutput of the policy model; increasing a time step by one; based on theincreased time step being less than a predetermined value, inputting theaction to the plurality of student models; and based on the increasedtime step being equal to the predetermined value, outputting the policymodel.
 16. The method of claim 9, further comprising: obtaining a policymodel by interacting with the ensemble student model; providing thepolicy model to each of the plurality of base stations for a policyevaluation of the policy model; and based on a training continue signalbeing received from at least one of the plurality of base stations as aresult of the policy evaluation, updating the ensemble student model andthe policy model, wherein the training continue signal is provided asthe feedback information, and indicates that a reward obtained from theensemble student model is less than a reward obtained from an existingload balancing model by a predetermined margin or more.
 17. Anon-transitory computer-readable storage medium storing a program thatis executable by at least one processor to perform a method forobtaining a load balancing artificial intelligence (AI) model for aplurality of base stations in a communication system, the methodcomprising: obtaining a plurality of teacher models based on a pluralityof traffic data sets collected from the plurality of base stations,respectively; obtaining a plurality of student models based on knowledgedistillation from the plurality of teacher models; obtaining an ensemblestudent model by ensembling the plurality of student models;transmitting the ensemble student model to the plurality of basestations, respectively; receiving feedback information of the ensemblestudent model from the plurality of base stations, and updating theensemble student model based on the received feedback information. 18.The non-transitory computer-readable storage medium of claim 17, whereineach of the plurality of teacher models comprises a state transitionmodel and a reward transition model that are trained based onstate-action-reward trajectories that are collected from the pluralityof base stations, wherein the state transition model is configured tooutput a predicted next state based on an action taken in a currentstate, and wherein the reward transition model is configured to output apredicted reward based on the action taken in the current state.
 19. Thenon-transitory computer-readable storage medium of claim 17, wherein theobtaining the plurality of student models based on knowledgedistillation from the plurality of teacher models, comprises: computinga ground-truth loss based on a difference between a ground-truth valueand a prediction of each of the plurality of student models; computing aknowledge distillation loss based on a difference between a teacherprediction of the plurality of teacher models and a student predictionof the plurality of student models; computing an aggregated loss thatcombines the ground-truth loss and the knowledge distillation loss; andtraining the plurality of student models by minimizing or converging theaggregated loss.
 20. The non-transitory computer-readable storage mediumof claim 17, wherein the obtaining the ensemble student model comprises:obtaining state-reward pairs from the plurality of student models;computing an average of the state-reward pairs; inputting the average ofthe state-reward pairs to an agent policy model to obtain an action asan output of the agent policy model; increasing a time step by one;based on the increased time step being less than a predetermined value,inputting the action to the plurality of student models; and based onthe increased time step being equal to the predetermined value,outputting the agent policy model.